Video Annotation and Retrieval with Pictorially Enriched Ontologies

Classifying video elements according to some predefined ontology of the video content domain is a typical way to perform video annotation. Ontologies are defined by establishing relationships between linguistic terms that specify domain concepts at different abstraction levels. However, although linguistic terms are appropriate to distinguish event and object categories, they are inadequate when they must describe specific or complex patterns of events or video entities. Instead, in these cases, pattern specifications can be better expressed using visual prototypes, either images or video clips, that capture the essence of the event or entity. Furthermore, video digital library require annotation at the level of pattern specification in order to retrieve multimedia content according to specific user preferences and high level semantic content description. Therefore enhanced ontologies, that include both visual and linguistic concepts, can be useful to support video annotation up to the level of detail of pattern specification. This paper presents algorithms and techniques that employ enriched ontologies for video annotation and retrieval, and discusses a solution for their implementation for the soccer video domain. Usage of reasoning on the ontology is shown, to perform higher-level annotation of the clips using the domain knowledge and to create complex queries that comprise visual prototypes of actions, their temporal evolution and relations.